SPARK-1093: Annotate developer and experimental API's #274

pwendell · 2014-03-31T00:38:44Z

This patch marks some existing classes as private[spark] and adds two types of API annotations:

EXPERIMENTAL API = experimental user-facing module
DEVELOPER API - UNSTABLE = developer-facing API that might change

There is some discussion of the different mechanisms for doing this here:
https://issues.apache.org/jira/browse/SPARK-1081

I was pretty aggressive with marking things private. Keep in mind that if we want to open something up in the future we can, but we can never reduce visibility.

A few notes here:

In the past we've been inconsistent with the visiblity of the X-RDD classes. This patch marks them private whenever there is an existing function in RDD that can directly creat them (e.g. CoalescedRDD and rdd.coalesce()). One trade-off here is users can't subclass them.
Noted that compression and serialization formats don't have to be wire compatible across versions.
Compression codecs and serialization formats are semi-private as users typically don't instantiate them directly.
Metrics sources are made private - user only interacts with them through Spark's reflection

AmplabJenkins · 2014-03-31T00:42:22Z

Merged build triggered. Build is starting -or- tests failed to complete.

AmplabJenkins · 2014-03-31T00:42:28Z

Merged build started. Build is starting -or- tests failed to complete.

AmplabJenkins · 2014-03-31T01:39:03Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-03-31T01:39:03Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13589/

mateiz · 2014-04-03T00:41:17Z

Hey Patrick, a few comments:

Should Logging become private[spark]? I think we looked at it before and there was some weird reason why we didn't do it, not sure what.
The badges unfortunately display weirdly in summary pages. For example look at this one for the root package. The first badge there is actually for Aggregator, but it displays on the bottom of its line, which makes it look like it might belong to the next element.
It might be good to give these different colors, e.g. something blue for Experimental API.

mateiz · 2014-04-03T00:54:03Z

Some other classes that may need to be annotated:

InterruptibleIterator -- probably private[spark]
RDD.*Approx (e.g. countApprox), and similar methods in JavaRDD, DoubleRDD, PairRDD -- experimental
RDD.mapPartitionsWithContext -- developer
JobLogger -- developer (I also saw it's deprecated)
scheduler.SchedulingMode -- private[spark]
scheduler.SplitInfo -- either private[spark] or developer (if it appears in events)
storage.StoragePerfTester -- should probably just be moved to the Spark Tools project
util.Vector -- deprecate this, MLlib will have its own, better ones
util.BoundedPriorityQueue -- any reason this is not just private[spark]?
SimpleFutureAction -- developer
SparkContext.addSparkListener -- developer
SparkContext.runApproximateJob -- developer
SparkContext.runJob -- developer
SparkContext.submitJob -- experimental
SparkContext.warnSparkMem -- probably need to be private
broadcast.BroadcastFactory -- developer
RDD.compute, dependencies, iterator and such should probably also be developer

mateiz · 2014-04-03T00:56:56Z

BTW to fix the floating badge problem, you might do the following: change code like this:

  <span class="badge badge-red" style="float: right;">DEVELOPER API - UNSTABLE</span>

   Represents a one-to-one dependency between partitions of the parent and child RDDs.

To this:

  <span class="badge badge-red" style="float: right;">DEVELOPER API - UNSTABLE
  </span>Represents a one-to-one dependency between partitions of the parent and child RDDs.

I believe Scaladoc includes only the first sentence it finds, so this might make it include both the text and the floating span. It might also not work though. But the reason those are displaying weirdly is partly that the first line there is ignored.

rxin · 2014-04-03T08:56:20Z

core/src/main/scala/org/apache/spark/rdd/CoGroupedRDD.scala

You might want to relax this since I don't think the user cannot construct Product2 version of the CoGroupedRDD in PairRDDFunctions

AmplabJenkins · 2014-04-05T23:52:23Z

Build triggered.

AmplabJenkins · 2014-04-05T23:52:32Z

Build started.

AmplabJenkins · 2014-04-05T23:54:35Z

Build finished.

AmplabJenkins · 2014-04-05T23:54:35Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13806/

AmplabJenkins · 2014-04-05T23:57:26Z

Build triggered.

AmplabJenkins · 2014-04-05T23:57:32Z

Build started.

AmplabJenkins · 2014-04-06T00:00:20Z

Build finished.

mengxr · 2014-04-09T05:30:51Z

That is the artifact name. Java uses annotation: http://docs.oracle.com/javase/7/docs/api/java/lang/annotation/Documented.html

pwendell · 2014-04-09T05:32:55Z

Okay - I'll change this to annotation

AmplabJenkins · 2014-04-09T05:37:23Z

Build triggered.

AmplabJenkins · 2014-04-09T05:37:30Z

Build started.

AmplabJenkins · 2014-04-09T06:34:06Z

Build finished. All automated tests passed.

AmplabJenkins · 2014-04-09T06:34:06Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13931/

Conflicts: core/src/main/scala/org/apache/spark/scheduler/JobResult.scala core/src/main/scala/org/apache/spark/storage/StorageUtils.scala core/src/main/scala/org/apache/spark/util/TimeStampedHashMap.scala sql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala

AmplabJenkins · 2014-04-09T07:22:23Z

Merged build triggered.

AmplabJenkins · 2014-04-09T07:22:31Z

Merged build started.

AmplabJenkins · 2014-04-09T08:01:10Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-04-09T08:01:10Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13939/

pwendell · 2014-04-09T08:16:26Z

Just merged this.

pwendell added 6 commits March 30, 2014 15:33

Initial pass

9d48cbf

Using better labels

42c1f09

More visiblity clean-up

5a76ed6

Another naming change

d86818e

Marking GraphX as alpha

cf23ec6

Small fixes

542a736

rxin reviewed Apr 3, 2014
View reviewed changes

pwendell added 3 commits April 5, 2014 13:54

Don't use floats

2f706f1

Code review feedback

cd7a465

Formatting and coloring of badges

1ed27d2

Style fixes

8452309

spark.annotations -> spark.annotation

042c803

asfgit closed this in 87bd1f9 Apr 9, 2014

peter-toth mentioned this pull request Jun 21, 2020

[SPARK-29375][SPARK-28940][SPARK-32041][SQL] Whole plan exchange and subquery reuse #28885

Closed

SPARK-1093: Annotate developer and experimental API's #274

SPARK-1093: Annotate developer and experimental API's #274

Uh oh!

Conversation

pwendell commented Mar 31, 2014

Uh oh!

AmplabJenkins commented Mar 31, 2014

Uh oh!

AmplabJenkins commented Mar 31, 2014

Uh oh!

AmplabJenkins commented Mar 31, 2014

Uh oh!

AmplabJenkins commented Mar 31, 2014

Uh oh!

mateiz commented Apr 3, 2014

Uh oh!

mateiz commented Apr 3, 2014

Uh oh!

mateiz commented Apr 3, 2014

Uh oh!

rxin Apr 3, 2014

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Apr 5, 2014

Uh oh!

AmplabJenkins commented Apr 5, 2014

Uh oh!

AmplabJenkins commented Apr 5, 2014

Uh oh!

AmplabJenkins commented Apr 5, 2014

Uh oh!

AmplabJenkins commented Apr 5, 2014

Uh oh!

AmplabJenkins commented Apr 5, 2014

Uh oh!

AmplabJenkins commented Apr 6, 2014

Uh oh!

mengxr commented Apr 9, 2014

Uh oh!

pwendell commented Apr 9, 2014

Uh oh!

AmplabJenkins commented Apr 9, 2014

Uh oh!

AmplabJenkins commented Apr 9, 2014

Uh oh!

AmplabJenkins commented Apr 9, 2014

Uh oh!

AmplabJenkins commented Apr 9, 2014

Uh oh!

AmplabJenkins commented Apr 9, 2014

Uh oh!

AmplabJenkins commented Apr 9, 2014

Uh oh!

AmplabJenkins commented Apr 9, 2014

Uh oh!

AmplabJenkins commented Apr 9, 2014

Uh oh!

pwendell commented Apr 9, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants